Protocol Engineering for Protection against Denial-of-Service Attacks

Denial-of-service attacks (DoS) and distributed denial-of-service attacks (DDoS) attempt to temporarily disrupt users or computer resources to cause service unavailability to legitimate users in the internetworking system. The most common type of DoS attack occurs when adversaries flood a large amount of bogus data to interfere or disrupt the service on the server. The attack can be either a single-source attack, which originates at only one host, or a multi-source attack, in which multiple hosts coordinate to flood a large number of packets to the server. Cryptographic mechanisms in authentication schemes are an example approach to help the server to validate malicious traffic. Since authentication in key establishment protocols requires the verifier to spend some resources before successfully detecting the bogus messages, adversaries might be able to exploit this flaw to mount an attack to overwhelm the server resources. The attacker is able to perform this kind of attack because many key establishment protocols incorporate strong authentication at the beginning phase before they can identify the attacks. This is an example of DoS threats in most key establishment protocols because they have been implemented to support confidentiality and data integrity, but do not carefully consider other security objectives, such as availability. The main objective of this research is to design denial-of-service resistant mechanisms in key establishment protocols. In particular, we focus on the design of cryptographic protocols related to key establishment protocols that implement client puzzles to protect the server against resource exhaustion attacks. Another objective is to extend formal analysis techniques to include DoSresistance. Basically, the formal analysis approach is used not only to analyse and verify the security of a cryptographic scheme carefully but also to help in the design stage of new protocols with a high level of security guarantee. In this research, we focus on an analysis technique of Meadows’ cost-based framework, and we implement DoS-resistant model using Coloured Petri Nets. Meadows’ iii cost-based framework is directly proposed to assess denial-of-service vulnerabilities in the cryptographic protocols using mathematical proof, while Coloured Petri Nets is used to model and verify the communication protocols using interactive simulations. In addition, Coloured Petri Nets are able to help the protocol designer to clarify and reduce some inconsistency of the protocol specification. Therefore, the second objective of this research is to explore vulnerabilities in existing DoS-resistant protocols, as well as extend a formal analysis approach to our new framework for improving DoS-resistance and evaluating the performance of the new proposed mechanism. In summary, the specific outcomes of this research include following results; 1. A taxonomy of denial-of-service resistant strategies and techniques used in key establishment protocols; 2. A critical analysis of existing DoS-resistant key exchange and key establishment protocols; 3. An implementation of Meadows’s cost-based framework using Coloured Petri Nets for modelling and evaluating DoS-resistant protocols; and 4. A development of new efficient and practical DoS-resistant mechanisms to improve the resistance to denial-of-service attacks in key establishment protocols.